A private web server will not respond to requests from public search engines.

Overview

Finding ID	Version	Rule ID	IA Controls	Severity
V-2260	WG310	SV-2260r1_rule		Low

Description
Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

Description

Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

STIG	Date
IIS 7.0 Server STIG	2019-03-22

Details

Check Text ( C-29395r1_chk )
Interview the SA to determine what type of restriction from public search engines are in place. If no means of restriction is in place (e.g. userid and password, domain or IP restriction, user PKI certificate), or a robots.txt file is not in use, this is finding. If a robots.txt files is used it must contain the following lines, if not then this is a finding. User-agent: * Disallow: /

Fix Text (F-26865r1_fix)
Establish a means to restrict search engines on the private web site.